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Abstract 

This  paper  develops  new  theory  and  algorithms  to  recover  signals  that  are 
approximately  sparse  in  some  general  (i.e. ,  basis,  frame,  over-complete,  or  in¬ 
complete)  dictionary  but  corrupted  by  a  combination  of  measurement  noise 
and  interference  having  a  sparse  representation  in  a  second  general  dictionary. 
Particular  applications  covered  by  our  framework  include  the  restoration  of 
signals  impaired  by  impulse  noise,  narrowband  interference,  or  saturation, 
as  well  as  image  in-painting,  super-resolution,  and  signal  separation.  We  de¬ 
velop  efficient  recovery  algorithms  and  deterministic  conditions  that  guaran¬ 
tee  stable  restoration  and  separation.  Two  application  examples  demonstrate 
the  efficacy  of  our  approach. 

Keywords:  Sparse  signal  recovery,  signal  restoration,  signal  separation, 
deterministic  recovery  guarantee,  basis-pursuit  denoising 


1.  Introduction 

We  investigate  the  problem  of  recovery  of  the  coefficient  vector  x  G  CNa 
from  the  corrupted  M-dimensional  observations 

z  =  Ax  +  Be  +  n,  (1) 

where  A  G  CMxNa  and  B  G  <CMxNb  are  general  (basis,  frame,  over-complete, 
or  in-complete)  deterministic  dictionaries,  i.e.,  matrices  whose  columns  have 
unit  Euclidean  (or  £ 2 )  norm.  The  vector  x  is  assumed  to  be  approximately 
sparse,  i.e.,  its  main  energy  (in  terms  of  the  sum  of  absolute  values)  is  con¬ 
centrated  in  only  a  few  entries.  The  M-dimensional  signal  vector  is  defined 
as  y  =  Ax.  The  vector  e  G  CNa  represents  interference  and  is  assumed  to  be 
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perfectly  sparse,  i.e.,  only  a  few  entries  are  nonzero,  and  n  €  CM  corresponds 
to  measurement  noise.  Apart  from  the  bound  ||n||2  <  e,  the  measurement 
noise  is  allowed  to  be  arbitrary.  The  interference  and  noise  components  e 
and  n  are  allowed  to  depend  on  the  vector  x  and/or  the  dictionary  A. 

The  setting  (1)  also  allows  us  to  study  signal  separation,  i.e.,  the  separa¬ 
tion  of  two  distinct  features  Ax  and  Be  from  the  noisy  observation  z.  Here, 
the  vector  e  in  (1)  is  also  allowed  to  be  approximately  sparse  and  is  used 
to  represent  a  second  desirable  feature  (rather  than  undesired  interference). 
Signal  separation  amounts  to  simultaneously  recovering  the  vectors  x  and  e 
from  the  noisy  measurement  z  followed  by  computation  of  the  individual 
signal  features  Ax  and  Be. 

1.1.  Applications  for  the  model  (1) 

Both  the  recovery  and  separation  problems  outlined  above  feature  promi¬ 
nently  in  numerous  applications  (see  [1-17]  and  references  therein),  including: 

•  Impulse  noise:  The  recovery  of  approximately  sparse  signals  corrupted 
by  impulse  noise  [13]  corresponds  to  recovery  of  x  from  (1)  by  setting 
B  =  I  m  and  associating  the  interference  e  with  the  impulse-noise  vec¬ 
tor.  Practical  examples  include  restoration  of  audio  signals  impaired 
by  click/pop  noise  [1,  2]  and  reading  out  from  unreliable  memory  [14]. 

•  Narrowband  interference:  Audio,  video,  or  communication  signals  are 
often  corrupted  by  narrowband  interference.  A  particular  example 
is  electric  hum,  which  typically  occurs  in  improperly  designed  audio 
equipment.  Such  impairments  naturally  exhibit  a  sparse  representation 
in  the  frequency  domain,  which  amounts  to  setting  B  to  the  Fourier 
matrix. 

•  Saturation:  Non-linearities  in  amplifiers  may  result  in  signal  saturation, 
e.g.,  [7,  16,  17].  Here,  instead  of  the  signal  vector  y  of  interest,  one 
observes  a  saturated  (or  clipped)  version  z  =  y  +  e  +  n,  where  the 
nonzero  entries  of  e  correspond  to  the  difference  between  the  saturated 
signal  and  the  original  signal  y .  The  noise  vector  n  can  be  used  to  model 
residual  errors  that  are  not  captured  by  the  interference  component  Be. 

•  Super-resolution  and  in-painting:  In  super-resolution  [3,  15]  and  in¬ 
painting  [6,  8-10]  applications,  only  a  subset  of  the  entries  of  the  (full- 
resolution)  signal  vector  y  =  Ax  is  available.  With  (1),  the  interference 
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vector  e  accounts  for  the  missing  parts  of  the  signal,  i.e.,  the  locations 
of  the  nonzero  entries  of  e  correspond  to  the  missing  entries  in  y  and 
are  set  to  some  arbitrary  value.  The  missing  parts  of  y  are  then  filled 
in  by  recovering  x  from  z  =  Ax  +  e  +  n  followed  by  computation  of 
the  (full-resolution)  signal  vector  y  =  Ax. 

•  Signal  separation:  The  framework  (1)  can  be  used  to  model  the  de¬ 
composition  of  signals  into  two  distinct  features.  Prominent  applica¬ 
tion  examples  are  the  separation  of  texture  from  cartoon  parts  in  im¬ 
ages  [4,  6]  and  separation  of  neuronal  calcium  transients  from  smooth 
signals  caused  by  astrocytes  in  calcium  imaging  [5].  In  both  applica¬ 
tions,  A  and  B  are  chosen  such  that  each  feature  can  be  represented 
by  approximately  sparse  vectors  in  one  dictionary.  Signal  separation 
then  amounts  to  simultaneously  extracting  x  and  e  from  z,  where  Ax 
and  Be  represent  the  individual  features. 

In  almost  all  of  the  applications  outlined  above,  a  predetermined  (and 
possibly  optimized)  dictionary  pair  A  and  B  is  used.  It  is  therefore  of  signif¬ 
icant  practical  interest  to  identify  the  fundamental  limits  on  the  performance 
of  restoration  or  separation  from  the  model  (1)  for  the  deterministic  setting, 
i.e.,  assuming  no  randomness  in  the  dictionaries,  the  signal,  interference,  or 
the  noise  vector.  Deterministic  recovery  guarantees  for  the  special  case  of 
perfectly  sparse  vectors  x  and  e  and  no  measurement  noise  have  been  stud¬ 
ied  in  [12,  18].  The  results  in  [12,  18]  rely  on  an  uncertainty  relation  for 
pairs  of  general  dictionaries  and  depend  on  the  number  of  nonzero  entries 
of  x  and  e,  on  the  coherence  parameters  of  the  dictionaries  A  and  B,  and 
on  the  amount  of  prior  knowledge  on  the  support  of  the  signal  and  interfer¬ 
ence  vector.  However,  the  algorithms  and  proof  techniques  used  in  [12,  18] 
cannot  be  adapted  for  the  general  (and  practically  more  relevant)  setting 
formulated  in  (1),  which  features  approximately  sparse  signals  and  additive 
measurement  noise. 

1.2.  Contributions 

In  this  paper,  we  generalize  the  recovery  guarantees  of  [12,  18]  to  the 
framework  (1)  detailed  above.  In  particular,  we  provide  novel,  computation¬ 
ally  efficient  restoration  and  separation  algorithms,  and  derive  corresponding 
recovery  guarantees  for  the  deterministic  setting.  Our  guarantees  depend  in 
a  natural  way  upon  the  number  of  dominant  nonzero  entries  of  x  and  e,  on 
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the  coherence  parameters  of  the  dictionaries  A  and  B,  and  on  the  Euclidean 
norm  of  the  measurement  noise.  Our  results  also  depend  on  the  amount  of 
knowledge  on  the  location  of  the  dominant  entries  available  prior  to  recov¬ 
ery.  In  particular,  we  investigate  the  following  cases:  1)  The  locations  of 
the  dominant  entries  of  the  approximately  sparse  vector  x  and  the  support 
set  of  the  perfectly  sparse  interference  vector  e  are  known  (prior  to  recov¬ 
ery),  2)  only  the  support  set  of  the  interference  vector  e  is  known,  and  3)  no 
support-set  knowledge  about  x  and  e  is  available.  Moreover,  we  present  new 
coherence-based  bounds  on  the  restricted  isometry  constants  (RICs)  for  the 
cases  2)  and  3),  which  we  then  use  to  derive  alternative  recovery  conditions 
using  the  restricted  isometry  property  (RIP)  framework.  We  provide  a  com¬ 
parison  to  the  recovery  conditions  for  perfectly  sparse  signals  and  noiseless 
measurements  presented  in  [12,  18].  Finally,  we  demonstrate  the  efficacy  of 
our  approach  with  two  representative  applications:  Restoration  of  audio  sig¬ 
nals  that  are  impaired  by  a  mixture  of  impulse  noise  and  Gaussian  noise,  and 
removal  of  scratches  from  old  photographs. 

1.3.  Notation 

Lowercase  and  uppercase  boldface  letters  stand  for  column  vectors  and 
matrices,  respectively.  The  transpose,  conjugate  transpose,  and  (Moore- 
Penrose)  pseudo-inverse  of  the  matrix  M  are  denoted  by  M7 ,  M77,  and 
=  (MhM)~  Mh,  respectively.  The  kth  entry  of  the  vector  m  is  [m]*,, 
and  the  kth  column  of  M  is  m*.  and  the  entry  in  the  kth  row  and  f'th  col¬ 
umn  is  designated  by  The  M  x  M  identity  matrix  is  denoted  by 

I m  and  the  M  x  N  all  zeros  matrix  by  OmxN-  The  Euclidean  (or  £2)  norm 
of  the  vector  x  is  denoted  by  ||x||2,  ||x||  1  =  JOfc|[x]fe|  stands  for  the  tj-norm 
of  x,  and  ||x||0  designates  the  number  of  nonzero  entries  of  x.  The  spec¬ 
tral  norm  of  the  matrix  M  is  ||M||2  =  A/Amax(M//M),  where  the  minimum 
and  maximum  eigenvalue  of  a  positive-semidehnite  matrix  M  are  denoted 

by  Amin(M)  and  Amax(M),  respectively.  ||M||F  =  [M]mP  stands  for 

the  Frobenius  matrix  norm.  Sets  are  designated  by  upper-case  calligraphic 
letters.  The  cardinality  of  the  set  T  is  \T\  and  the  complement  of  a  set  S 
in  some  superset  T  is  denoted  by  Sc.  The  support  set  of  the  vector  m,  i.e., 
the  index  set  corresponding  to  the  nonzero  entries  of  m,  is  designated  by 
supp(m).  We  define  the  M  x  M  diagonal  (projection)  matrix  for  the  set 
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S  C  {1, . . . ,  M }  as  follows: 

r  _  f  1,  k  =  l  and  k  G  S 
[  s\k,e  |  g,  otherwise, 

and  nif  =  P-7-m.  The  matrix  M7-  is  obtained  from  M  by  retaining  the 
columns  of  M  with  indices  in  T  and  the  \T\ -dimensional  vector  [m]7-  is 
obtained  analogously.  For  x  G  M,  we  set  [:c]+  =  max{x,  ()}. 


1.4 ■  Outline  of  the  paper 

The  remainder  of  the  paper  is  organized  as  follows.  In  Section  2,  we 
briefly  summarize  the  relevant  prior  art.  Our  new  recovery  algorithms  and 
corresponding  recovery  guarantees  are  presented  in  Section  3.  An  alterna¬ 
tive  set  of  recovery  guarantees  obtained  through  the  RIP  framework  and  a 
comparison  to  existing  recovery  guarantees  are  provided  in  Section  4.  The 
application  examples  are  shown  in  Section  5,  and  we  conclude  in  Section  6. 


2.  Relevant  Prior  Art 

In  this  section,  we  review  the  relevant  prior  art  in  recovering  sparse  sig¬ 
nals  from  noiseless  and  noisy  measurements  in  the  deterministic  setting  and 
summarize  the  existing  guarantees  for  recovery  of  sparsely  corrupted  signals. 

2.1.  Recovery  of  perfectly  sparse  signals  from  noiseless  measurements 

Recovery  of  a  vector  x  G  CNa  from  the  noiseless  observations  y  =  Ax 
with  A  over-complete  (i.e.,  M  <  Na)  corresponds  to  solving  an  underde¬ 
termined  system  of  linear  equations,  which  is  well-known  to  be  ill-posed. 
However,  assuming  that  x  is  perfectly  sparse  (i.e.,  that  only  small  number  of 
its  entries  are  nonzero)  enables  us  to  uniquely  recover  x  by  solving 

(PO)  minimize  ||x||0  subject  to  y  =  Ax. 

Unfortunately,  PO  has  a  prohibitive  (combinatorial)  computational  complex¬ 
ity,  even  for  small  dimensions  Na.  One  of  the  most  popular  and  computation¬ 
ally  tractable  alternative  to  solving  PO  is  basis  pursuit  (BP)  [19-24],  which 
corresponds  to  the  convex  program 

(BP)  minimize  ||x|| x  subject  to  y  =  Ax. 
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Recovery  guarantees  for  PO  and  BP  are  usually  expressed  in  terms  of  the 
sparsity  level  nx  =  ||x||0  and  the  coherence  parameter  of  the  dictionary  A, 
which  is  defined  as 

Ha  =  max  |afaJ  . 

k/,k^e 1  1 

Specifically,  a  sufficient  condition  for  x  to  be  the  unique  solution  of  PO  and 
for  BP  to  deliver  this  solution1  is  [21,  22,  24] 

n'<Ki+^) 

2.2.  Recovery  of  approximately  sparse  signals  from  noisy  measurements 
For  the  case  of  bounded  (otherwise  arbitrary)  measurement  noise,  i.e., 
z  =  Ax  +  n  with  ||n||2  <  e,  recovery  guarantees  based  on  the  coherence 
parameter  pa  were  developed  in  [27-31].  The  corresponding  recovery  con¬ 
ditions  mostly  treat  the  case  of  perfectly-sparse  signals,  i.e.,  where  only  a 
small  fraction  of  the  entries  x  are  nonzero.  In  almost  all  practical  applica¬ 
tions,  however,  only  a  few  entries  of  x  are  actually  zero.  Nevertheless,  many 
real-world  signals  exhibit  the  property  that  most  of  the  signal’s  energy  (in 
terms  of  the  sum  of  absolute  values)  is  concentrated  in  only  a  few  entries. 
We  refer  to  this  class  of  signals  as  approximately  sparse  in  the  remainder  of 
the  paper.  For  such  signals,  the  support  set  associated  to  the  best  nx-sparse 
approximation  (in  f^-norm)  corresponds  to 

X  =  supprJx)  =  arg  min  ||x  -  x^ ||x  , 

X 

where  the  set  T,nx  contains  all  support  sets  of  size  nx  corresponding  to  per¬ 
fectly  nx-sparse  vectors  having  the  same  dimension  as  x.  A  particular  sub¬ 
class  of  approximately  sparse  signals  is  the  set  of  compressible  signals,  whose 
approximation  error  decreases  according  to  a  power  law  with  exponent  s  >  0 
as  nx  increases  [32], 

The  following  theorem  provides  a  sufficient  condition  for  which  a  suitably 
modified  version  of  BP,  known  as  BP  denoising  (BPDN)  [19],  stably  recovers 
an  approximately  sparse  vector  x  from  the  noisy  observation  z. 

lrThe  condition  (2)  also  ensures  perfect  recovery  using  orthogonal  matching  pur¬ 
suit  (OMP)  [24-26],  which  is,  however,  not  further  investigated  in  this  paper. 


(2) 
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Theorem  1  (BP  denoising  [31,  Thm.  2.1]).  Let  z  =  Ax  +  n,  ||n  1 1 2  —  £> 

and  X  =  suppn  (x).  If  (2)  is  met,  then  the  solution  x  of  the  convex  program 

(BPDN)  minimze  ||x||  1  subject  to  ||z  —  Ax||2  <  g 
with  e  <  7]  satisfies 


||x-x||2  <  C0(£  +  rj)  +  Ci||x  -  x^^  ,  (3) 

where  both  (non-negative)  constants  Co  and  C\  depend  on  pa  and  nx. 

PROOF.  The  proof  in  [31,  Thm.  2.1]  is  detailed  for  perfectly  sparse  vectors 
only.  Since  some  of  the  proofs  presented  in  the  remainder  of  the  paper  are  for 
approximately  sparse  signals  and  noisy  measurements,  we  present  the  general 
case  in  Appendix  A. 

We  emphasize  that  perfect  recovery  of  x  is,  in  general,  impossible  in  the 
presence  of  bounded  (but  otherwise  arbitrary)  measurement  noise  n.  In  the 
remainder  of  the  paper,  we  consider  stable  recovery  instead,  i.e.,  in  a  sense 
that  the  ^-norm  of  the  difference  between  the  estimate  x  and  the  ground 
truth  x  is  bounded  from  above  by  the  £2-norm  of  the  noise  ||n||2  and  the 
best  nx-sparse  approximation  in  G-norm  sense,  i.e.,  as  in  (3).  We  finally 
note  that  Theorem  1  generalizes  the  results  for  noiseless  measurements  and 
perfectly  sparse  signals  in  [21,  22,  24]  using  BP  (cf.  Section  2.1).  Specifically, 
for  || n|| 2  =  0  and  ||x  — x*^  =  0,  BPDN  with  p  =  0  corresponds  to  BP 
and  (3)  results  in  ||x  —  x||2  =  0,  which  ensures  perfect  recovery  of  the  vector  x 
whenever  (2)  is  met. 

2.3.  Recovery  guarantees  for  perfectly  sparse  signals  from  sparsely  corrupted 
and  noiseless  measurements 

A  large  number  of  restoration  and  separation  problems  occurring  in  prac¬ 
tice  can  be  formulated  as  sparse  signal  recovery  from  sparsely  corrupted 
signals  using  the  input-output  relation  (1).  Special  cases  of  the  general 
model  (1)  were  studied  in  [7,  11-13,  18,  33-37]. 

Probabilistic  recovery  guarantees.  Recovery  guarantees  for  the  probabilistic 
setting  (i.e.,  recovery  of  x  is  guaranteed  with  high  probability)  for  random 
Gaussian  matrices,  which  are  of  particular  interest  for  applications  based  on 
compressive  sensing  (CS),  were  reported  in  [7,  11,  35,  37].  Similar  results 
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for  randomly  sub-sampled  unitary  matrices  A  were  developed  in  [36].  The 
problem  of  sparse  signal  recovery  from  a  particular  nonlinear  measurement 
process  in  the  presence  of  impulse  noise  was  considered  in  [13],  and  prob¬ 
abilistic  results  for  signal  detection  based  on  £ i -norm  minimization  in  the 
presence  of  impulse  noise  was  investigated  in  [34],  In  the  remainder  of  the 
paper,  however,  we  will  consider  the  deterministic  setting  exclusively. 

Deterministic  recovery  guarantees.  Recovery  guarantees  in  the  deterministic 
setting  for  noiseless  measurements  and  signals  being  perfectly  sparse,  i.e., 
the  model  z  =  Ax  +  Be,  were  studied  in  [12,  18,  33].  In  [33],  it  was  shown 
that  when  A  is  the  Fourier  matrix,  B  =  I m  and  when  the  support  set  of 
the  interference  e  is  known,  perfect  recovery  of  x  is  possible  if  2 nxne  <  M, 
where  ne  =  ||e||0.  The  case  of  A  and  B  being  arbitrary  dictionaries  (whereas 
x  and  e  are  assumed  to  be  perfectly  sparse  and  for  noiseless  measurements) 
was  studied  for  different  cases  of  support-set  knowledge  in  [12,  18].  There, 
deterministic  recovery  guarantees  were  presented,  which  depended  upon  the 
number  of  nonzero  entries  nx  and  ne  in  x  and  e,  respectively,  and  on  the  co¬ 
herence  parameters  /aa  and  //j,  of  A  and  B,  as  well  as  on  the  mutual  coherence 
between  the  dictionaries  A  and  B,  which  is  defined  as 

/ im  =  max  laf  bJ  . 

k,£ 

A  summary  of  the  recovery  guarantees  presented  in  [12,  18]  (along  with  the 
novel  recovery  guarantees  presented  in  the  next  section)  is  given  in  Table  1, 
where,  for  the  sake  of  simplicity  of  exposition,  we  define  the  function 

f(u ,  v)  =  [1  -  na(u  -  1)]+  [1  -  nb(v  -  1)]+  . 

We  emphasize  that  the  results  presented  in  [12,  18]  are  for  perfectly  sparse 
and  noiseless  measurements  only,  and  furthermore,  the  algorithms  and  proof 
techniques  cannot  be  adapted  for  the  more  general  setting  proposed  in  (1).  In 
order  to  gain  insight  into  the  practically  more  relevant  case  of  approximately 
sparse  signals  and  noisy  measurements,  we  next  develop  new  restoration  and 
separation  algorithms  for  several  different  cases  of  support-set  knowledge  and 
provide  corresponding  recovery  guarantees.  Our  results  complement  those  in 
[12,  18]  (cf.  Table  1). 

3.  Main  Results 

We  now  develop  several  computationally  efficient  methods  for  restoration 
or  separation  under  the  model  (1)  and  derive  corresponding  recovery  condi- 


Table  1:  Summary  of  deterministic  recovery  guarantees  for  perfectly/approximately  sparse 
signals  that  are  corrupted  by  interference  in  the  absence/presence  of  measurement  noise. 


Support-set 

knowledge 

Recovery  condition 

Perfectly  sparse 
and  no  noise 

Approx,  sparse 
and  noise 

x  and  e 

nxnen2m  <  f{nx ,  ne ) 

[12,  Thm.  3] 

Theorem  2 

e  only 

2nxnepL2m  <  f(2nx,  ne) 

[12,  Thms.  4  and  5] 

Theorem  3 

None 

[18,  Eq.  12] a 

Eq.  13 

[18,  Thm.  3] 

Theorem  4 

“The  recovery  condition  is  valid  for  BP  and  OMP;  a  less  restrictive  condition  for  PO  is 
given  in  [18,  Thm.  2]. 

tions  that  guarantee  their  stability.  Our  recovery  guarantees  depend  on  the 
^2- norm  of  the  noise  vector  and  on  the  amount  of  knowledge  on  the  domi¬ 
nant  nonzero  entries  of  the  signal  and  noise  vectors.  Specifically,  we  consider 
the  following  three  cases:  1 )  Direct  restoration:  The  locations  of  the  entries 
corresponding  to  the  best  n^-sparse  approximation  of  x  and  the  support  set 
of  the  (perfectly  sparse)  interference  vector  e  are  known  prior  to  recovery,  2) 
BP  restoration:  Only  the  support  set  of  e  is  known,  3)  BP  separation:  No 
knowledge  about  x  and  e  is  available,  except  for  the  fact  that  each  vector 
exhibits  an  approximately  an  sparse  representation  in  either  A  or  B. 

3.1.  Direct  restoration:  Support-set  knowledge  of  x  and  e 

We  start  by  addressing  the  case  where  the  locations  of  the  dominant 
entries  (in  terms  of  absolute  value)  of  the  approximately  sparse  vector  x  and 
the  support  set  £  associated  with  the  perfectly  sparse  interference  vector  e 
are  known  prior  to  recovery.  This  scenario  is  relevant,  for  example,  in  the 
restoration  of  old  phonograph  records  [1,  2],  where  one  wants  to  recover  a 
bandlimited  signal  that  is  impaired  by  impulse  noise,  e.g.,  clicks  and  pops. 
The  occupied  frequency  band  of  phonograph  recordings  is  typically  known 
prior  to  recovery.  In  this  case,  one  may  assume  A  to  be  the  M-dimensional 
discrete  cosine  transform  (DCT)  matrix  and  B  =  I m-  The  locations  of  the 
clicks  and  pops,  i.e.,  the  support  set  £  =  supp(e),  can  be  determined  (prior 
to  recovery)  using  the  techniques  described  in  [2],  for  example. 

The  restoration  approach  considered  for  this  setup  is  as  follows.  Since  £ 
and  X  =  suppnJx)  are  both  known  prior  to  recovery,  we  start  by  projecting 
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the  noisy  observation  vector  z  onto  the  orthogonal  complement  of  the  range 
space  spanned  by  Be,  which  eliminates  the  sparse  corruptions  caused  by  the 
interference  e.  In  particular,  we  consider 

Rgz  =  R£(Ax  +  Be^-  +  n)  =  Re  Ax  +  Rfn,  (4) 

where  Rg  =  I m  ~  B^Bt  is  the  projector  onto  the  orthogonal  complement  of 
the  range  space  of  B^,  and  we  used  the  fact  that  R^Be^-  =  Omxi-  Next,  we 
separate  (4)  by  exploiting  the  fact  that  X  is  known 

RfZ  =  R£A(x*  +  x^c)  +  Rfn 

=  R^-A^fx]^-  +  R^-Ax^c  +  R^n 

and  isolate  the  dominant  entries  [x]*  as  follows: 

(R£AxyR£z  =  [x\x  +  (R^Ax)*Rj(Ax^c  +  n).  (5) 

In  the  case  where  both  vectors  xXc  and  n  are  equal  to  zero,  we  obtain 

(RfAx^RfZ  =  [x]*,  (6) 

and  therefore  the  dominant  entries  of  x  are  recovered  perfectly  by  this  ap¬ 
proach.  Note  that  conjugate  gradient  methods  (see,  e.g.,  [38])  offer  an  effi¬ 
cient  way  of  computing  (6). 

The  following  theorem  provides  a  sufficient  condition  for  (R^AyjlRf 
to  exist  and  for  which  the  vector  x  can  be  restored  stably  from  the  noisy 
measurement  z  using  the  direct  restoration  (DR)  procedure  outlined  above. 

Theorem  2  (Direct  restoration).  Let  z  =  Ax  +  Be  +  n  with  ||n  II 2  —  £> 

e  perfectly  ne-sparse  with  support  set  £,  and  X  =  suppn  (x).  Furthermore, 
assume  that  £  and  X  are  known  prior  to  recovery.  If 

nxnep2m  <  f(nx,ne),  (7) 

then  the  vector  x  computed  according  to 

(DR)  [x]*  =  (R^A^R^-z,  [x]*c  =  0|^c|xi 

with  R^  =  I m  ~  BfBt  satisfies 

||x-x||2  <  C'3£  +  C'4||x-x^||1, 

where  the  (non-negative)  constants  C3  and  C4  depend  on  the  coherence  pa¬ 
rameters  Pa,  Pb,  and  pm,  and  on  the  sparsity  levels  nx  and  ne. 


10 


PROOF.  The  proof  is  given  in  Appendix  B. 

Theorem  2  and  in  particular  (7)  provides  a  sufficient  condition  on  the 
number  nx  of  dominant  entries  of  x  for  which  DR  enables  the  stable  recovery 
of  x  from  z,  given  the  coherence  parameters  [ia ,  fib ,  arid  /im,  and  the  number 
of  sparse  corruptions  ne.  Specifically,  (7)  states  that  for  a  given  number  of 
sparse  corruptions  ne,  the  smaller  the  coherence  parameters  fia,  /p,,  and  fim , 
the  more  dominant  entries  of  x  can  be  recovered  stably  from  z.  The  case 
that  guarantees  the  recovery  of  the  largest  number  nx  of  dominant  entries  in 
x  is  when  A  and  B  are  orthonormal  bases  (ONBs)  (i.e.,  /ia  =  Pb  =  0)  that 
are  maximally  incoherent  (i.e.,  prn  =  l/\/M);  this  is  the  case  for  the  Fourier- 
identity  pair,  whence  the  recovery  condition  (7)  corresponds  to  nxne  <  M . 

The  recovery  guarantee  in  Theorem  2  generalizes  that  in  [12,  Thm.  3] 
to  approximately  sparse  signals  and  noisy  measurements.  In  particular,  for 
|| n|| 2  =  0  and  ||x  —  ^~x\\i  =  0,  we  have  e  =  0  and  that  DR  perfectly  recovers  x 
if  (7)  is  met.  Since  (7)  is  identical  to  the  condition  [12,  Thm.  3]  (cf.  Table  1) 
we  see  that  considering  approximately  sparse  signals  and  (bounded)  mea¬ 
surement  noise  does  not  result  in  a  more  restrictive  recovery  condition.  We 
finally  note  that  the  recovery  condition  in  (7)  was  shown  in  [12]  to  be  tight  for 
certain  comb  signals  in  the  case  where  A  is  the  Fourier  matrix  and  B  =  Im- 

3.2.  BP  restoration:  Support-set  knowledge  of  e  only 

Next,  we  find  conditions  guaranteeing  the  stable  recovery  in  the  setting 
where  the  support  set  of  the  interference  vector  e  is  known  prior  to  recov¬ 
ery.  A  prominent  application  for  this  setting  is  the  restoration  of  saturated 
signals  [7,  16].  Here,  no  knowledge  on  the  locations  of  the  dominant  en¬ 
tries  of  x  is  required.  The  support  set  £  of  the  sparse  interference  vector 
can,  however,  be  easily  identified  by  comparing  the  measured  signal  entries 
[z]j,  i  —  1, . . . ,  M,  to  a  saturation  threshold.  Further  application  examples 
for  this  setting  include  the  removal  of  impulse  noise  [1,  2,  14],  in-painting, 
and  super-resolution  [3,  8,  15]  of  signals  admitting  an  approximately  sparse 
representation  in  some  arbitrary  dictionary  A. 

The  recovery  procedure  for  this  case  is  as  follows.  Since  £  is  know  prior 
to  recovery,  we  may  recover  the  vector  x  by  projecting  the  noisy  observation 
vector  z  onto  the  orthogonal  complement  of  the  range  space  spanned  by  B^ 
(cf.  Section  3.1).  This  projection  eliminates  the  sparse  noise  and  leaves  us 
with  a  sparse  signal  recovery  problem  similar  to  that  in  Theorem  1.  In 
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particular,  we  consider  recovery  from 

Rgz  =  Rf(Ax  +  Be,*-  +  n)  =  RcAx  +  R^n,  (8) 

where  Rc  =  IM— B^Bjr,  and  we  used  the  fact  that  RcBoc  =  Omxi-  Note  that 
the  idea  of  projecting  the  observation  vector  onto  the  orthogonal  complement 
of  the  space  spanned  by  the  active  columns  of  B  was  put  forward  in  [7],  where 
the  special  case  of  B  being  an  ONB  was  considered.  The  following  theorem 
provides  a  sufficient  condition  that  guarantees  the  stable  restoration  of  the 
vector  x  from  (8). 


Theorem  3  (BP  restoration).  Let  z  =  Ax  +  Be  +  n  with  ||n  II 2  — 
Assume  e  to  be  perectly  ne-sparse  and  £  =  supp(e )  to  be  known  prior  to 
recovery.  Furthermore,  let  X  =  suppn  (x) .  If 

2nxnen2m  <  f(2nx,  ne),  (9) 


then  the  result  x  of  BP  restoration 


(BP-RES) 


minimize 


subject  to  ||  R^  (z  —  Ax)||  2  —  V 


with  Rf  =  I m  ~  BfBl  and  £  <  q  satisfies 


x||2  <  C5(e  +  rj)  +  C'6||x-x*||1, 


where  the  (non-negative)  constants  C§  and  C§  depend  on  the  coherence  pa¬ 
rameters  fia,  Hb,  and  Am,  and  on  the  sparsity  levels  nx  and  ne. 


PROOF.  The  proof  is  given  in  Appendix  C. 

The  recovery  condition  (9)  provides  a  sufficient  condition  on  the  number 
nx  of  dominant  entries  of  x,  for  which  BP-RES  can  stably  recover  x  from  z. 
The  condition  depends  on  the  coherence  parameters  pa,  Ab,  and  /im,  and  the 
number  of  sparse  corruptions  ne.  As  for  the  case  of  DR,  the  situation  that 
guarantees  that  the  largest  number  nx  of  dominant  coefficients  in  x  will  be 
recovered  stably  using  BP-RES,  is  when  A  and  B  are  maximally  incoherent 
ONBs.  In  this  situation,  the  recovery  condition  (9)  reduces  to  2 nxne  <  M, 
which  is  two  times  more  restrictive  than  that  for  DR  (see  [12|  for  a  detailed 
discussion  on  this  factor-of-two  penalty). 

The  following  observations  are  immediate  consequences  of  Theorem  3: 
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•  If  the  vector  x  is  perfectly  nx-sparse,  i.e.,  ||x  —  =  0,  and  no 

measurement  noise  is  present,  i.e.,  £  =  0,  then  BP-RES  using  rj  —  0 
perfectly  recovers  x  whenever  (9)  is  satisfied.  Note  that  two  different 
restoration  procedures  were  developed  for  this  particular  setting  in  [12, 
Thms.  4  and  5].  Both  methods  enable  perfect  recovery  under  exactly 
the  same  conditions  (cf.  Table  1).  Hence,  generalizing  the  recovery 
procedure  to  approximately  sparse  signals  and  measurement  noise  does 
not  incur  a  penalty  in  terms  of  the  recovery  condition. 

•  The  restoration  method  in  [12,  Thm.  5]  requires  a  column-normalization 
procedure  to  guarantee  perfect  recovery  under  the  condition  (9).  Since 
in  this  special  case,  BP-RES  (with  r)  =  0)  corresponds  to  BP,  Theo¬ 
rem  3  implies  that  this  normalization  procedure  is  not  necessary  for 
guaranteeing  perfect  recovery  under  (9).  Note,  however,  that  this  ob¬ 
servation  does  not  apply  to  orthogonal  matching  pursuit  (see  [39]  for 
more  details). 

We  finally  point  out  that  the  recovery  condition  in  (9)  was  shown  in  [12] 
to  be  tight  for  particularly-chosen  comb  signals  in  the  case  where  A  is  the 
Fourier  matrix  and  B  =  1m- 

3.3.  BP  separation:  No  knowledge  on  the  support  sets 

We  finally  consider  the  case  where  no  knowledge  about  the  support  sets 
of  the  approximately  sparse  vectors  x  and  e  is  available.  A  typical  appli¬ 
cation  scenario  is  signal  separation  [4,  6],  e.g.,  the  decomposition  of  audio, 
image,  or  video  signals  into  two  or  more  distinct  features,  i.e.,  in  a  part 
that  exhibits  an  approximately  sparse  representation  in  the  dictionary  A 
and  another  part  that  exhibits  an  approximately  sparse  representation  in  B. 
Decomposition  then  amounts  to  performing  simultaneous  recovery  of  x  and 
e  from  z  =  Ax  +  Be  +  n,  followed  by  computation  of  the  individual  signal 
features  according  to  Ax  and  Be.  The  main  idea  underlying  this  signal- 
separation  approach  is  to  rewrite  (1)  as 

z  =  Dw  +  n  (10) 

where  D  =  [  A  B  ]  is  the  concatenated  dictionary  of  A  and  B  and  the  stacked 
vector  w  =  [x3  e3  ]T.  Signal  separation  now  amounts  to  performing  BPDN 
on  (10)  for  recovery  of  w  from  z,  which  is  also  known  as  the  synthesis  separa¬ 
tion  problem  for  microlocal  analysis  (see  [40,  41]  and  the  references  therein). 
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A  straightforward  way  to  arrive  at  a  corresponding  deterministic  recovery 
guarantee  for  this  problem  is  to  consider  D  as  the  new  dictionary  with  the 
dictionary  coherence  defined  as 

Pd  =  max  I  df  dj  I  =  max  {pa,  pb,  pm\ .  (11) 


One  can  now  use  BPDN  to  recover  w  from  (10)  and  invoke  Theorem  1  with 
the  recovery  condition  in  (2),  resulting  in 


w  —  nx  +  ne  < 


(12) 


However,  it  is  important  to  realize  that  (12)  ignores  the  structure  underlying 
the  dictionary  D,  i.e.,  it  does  not  take  into  account  the  fact  that  D  is  a 
concatenation  of  two  dictionaries  that  are  characterized  by  the  coherence 
parameters  pa,  Pb,  and  pm.  Hence,  the  recovery  guarantee  (12)  does  not 
provide  insight  into  which  pairs  of  dictionaries  A  and  B  are  most  useful  for 
signal  separation.  The  following  theorem  takes  into  account  the  structure 
underlying  D,  enabling  us  to  gain  insight  into  which  pairs  of  dictionaries  A 
and  B  support  signal  separation. 


Theorem  4  (BP  separation).  Let  z  =  Dw  +  n7  with  D  =  [AB],  w  = 

[a7  eT]T,  and  ||n||2  <  e.  The  dictionary  D  is  characterized  by  the  coherence 
parameters  pa,  pb,  Pm,  and  Hd,  and  we  assume  pb  <  pa  without  loss  of 
generality.  Furthermore,  let  W  =  suppw(w) .  If 


w  <  max 


2(1  +  pa) 

Pa  +  2 Pd  +  yj Pa  T  p2m 


1  +  pd  ] 

2  Pd  J  ’ 


then  the  solution  w  of  BP  separation 


(BP-SEP) 


J  minimize  Hw^ 

(  subject  to  || z  —  Dw  II 2  —  ^7 


using  £  <  rj  satisfies 


(13) 


w  -  w||2  <  C7(e  +  rj)  +  C8||w  -  wwl^  , 


(14) 


with  the  (non-negative)  constants  C7  and  Cg. 
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PROOF.  The  proof  is  given  in  Appendix  D. 

The  recovery  condition  (13)  refines  that  in  (12);  in  particular,  considering 
the  two-ONB  setting  for  which  /ia  —  /if,  —  0  and  /im  =  fa.  In  this  case,  the 
straightforward  recovery  condition  (11)  corresponds  to  w  <  (1  +  l/fa)/2, 
whereas  the  one  for  BP  separation  (13)  is 

w  <  (15) 

3  l^d 

Hence,  (13)  guarantees  the  stable  recovery  for  a  larger  number  of  dominant 
entries  w  in  the  stacked  vector  w.  Recovery  guarantees  for  perfectly  sparse 
signals  and  noiseless  measurements  in  the  two-ONB  setting  were  developed 
in  [23,  24,  42] .  The  corresponding  recovery  condition  w  <  (V2  —  0.5) / fa  turns 
out  to  be  better  (i.e.,  less  restrictive)  than  the  recovery  condition  for  approxi¬ 
mately  sparse  signals  and  measurement  noise  provided  in  (15).  Whether  this 
behavior  is  a  fundamental  result  of  considering  approximately  sparse  signals 
and  noisy  measurements  or  is  an  artifact  of  the  proof  technique  remains  an 
open  research  problem. 

4.  Recovery  Guarantees  from  the  RIP-Framework  and  Comparison 

In  this  section,  we  develop  alternative  recovery  guarantees  for  BP-RES 
and  BP-SEP  using  results  from  the  restricted  isometry  property  (RIP)  frame¬ 
work.  We  furthermore  provide  a  comparison  with  the  (coherence-based)  re¬ 
covery  guarantees  obtained  in  the  previous  section  and  the  guarantees  for 
perfectly  sparse  signals  and  noiseless  measurements  presented  in  [12,  18]  (re¬ 
call  Table  1). 

4-1.  RIP- based  recovery  guarantees 

An  alternative  way  of  obtaining  deterministic  recovery  guarantees  for  ap¬ 
proximately  sparse  signals  and  measurement  noise,  i.e.,  for  z  =  Ax  +  n,  has 
been  developed  under  the  RIP  framework  [34,  43-47].  There,  the  dictio¬ 
nary  A  is  characterized  by  restricted  isometry  constants  (RICs)  instead  of 
the  coherence  parameter  pa . 

Definition  1  ([34]).  For  each  integer  nx  >  1,  the  RIC  5Ux  of  A  is  the 
smallest  number  such  that 

(1  -  <SnJ  IMI2  <  II Axll 2  <  (!  +  Snx)  IMI2  (16) 

holds  for  all  perfectly  nx-sparse  vectors  x. 
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The  corresponding  recovery  conditions,  which  guarantee  stable  recovery 
of  x  with  || n|| 2  <  e  using  BDPN,  are,  e.g.,  of  the  form  i)  $2nx  <  V%  —  1  [44]  or 
ii)  8Ux  <  0.307  [47].  The  main  issue  with  such  recovery  conditions  is  the  fact 
that  the  RICs  cannot  be  computed  efficiently,  in  general.  In  order  to  arrive 
at  recovery  conditions  that  are  explicit  in  the  number  of  nonzero  entries  nx , 
one  may  bound  the  RIC  (16)  from  above  using  the  coherence  parameter  pa 
as  [31,  45] 


<5nx  <  fia(nx  -  1)  (17) 

which  can  then  be  used  to  arrive  at  the  corresponding  recovery  conditions 
i)  nx  <  (1  +  (\/2  —  l)/na)/2  or  ii)  nx  <  1  +  0.307/pa.  Both  of  these  recov¬ 
ery  conditions  are  more  restrictive  than  that  in  (2).  Nevertheless,  recovery 
guarantees  obtained  through  the  RIP  framework  turn  out  to  be  useful  for 
deriving  probabilistic  recovery  conditions  (guaranteeing  recovery  with  high 
probability)  in  the  field  of  compressive  sensing  (CS)  [48,  49].  The  coherence- 
based  bound  in  (17)  is  useful  also  because  it  puts  limits  on  the  RIC  that  are 
explicit  in  nx . 


4-2.  Recovery  guarantees  for  sparsely  corrupted  signals 

We  next  provide  coherence-based  bounds  on  the  RIC  constants  for  BP 
restoration  and  BP  separaion,  and  derive  corresponding  alternative  recovery 
guarantees  using  results  obtained  in  the  RIP  framework  [44,  47]. 


Recovery  guarantee  for  BP  restoration.  As  a  byproduct  of  the  proof  for  BP- 
RES  detailed  in  Appendix  C,  the  following  coherence-based  upper  bound  on 
the  RIC  for  the  matrix  A  =  Rc  A  was  obtained: 


Lemma  5  (RIC  bound  for  A).  Let  A  =  Rg A  with  Rg  —  IM  —  B^Bt. 

For  each  integer  nx  >  1,  the  smallest  number  8Ux  such  that 


(1  —  &nx)  IMI2  < 


Ax 


<  (1+0  ||X||; 


holds  for  all  perfectly  nx-sparse  vectors  x  €  CA0,  is  bounded  from  above  by 


8 nx  —  RaiPx 


1)  + 


nxneRm 

[1  -  nb(ne  -  1)]  + 


(18) 
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Combining  the  bound  (18)  with  the  condition  5nx  <  0.307  from  [47]  en¬ 
ables  us  to  obtain  the  following  recovery  condition: 

n  <  (0-307  +  pa)[l  —  /ife(ne  -  1)]+  /igv 

pa[  1  -  Hb(ne~  1)]+  +  nen2m 

that  guarantees  the  stable  restoration  of  x  from  R^z  =  Ax  +  R^n  using  BP- 
RES.  We  emphasize  that  for  pa  <  1  —  2  ■  0.307  ~  0.386  the  condition  (19)  is 
more  restrictive  than  the  condition  (9),  which  reads 

1  (l  +  /iq)[l-Mne-l)]  + 

2  Ra[  1  -  Pb{ne  -  1)]+  +  nep2m 

Hence,  for  most  relevant  values  of  the  coherence  parameter  pa,  the  recovery 
condition  in  (9)  is  superior  (in  a  sense  of  enabling  the  recovery  of  a  larger 
number  nx  of  dominant  coefficients  in  x)  to  that  obtained  using  the  RIP 
framework. 


Recovery  guarantee  for  BP  separation.  During  the  proof  for  BP  separation 
detailed  in  Appendix  D,  the  following  coherence-based  upper  bound  on  the 
RIC  for  the  concatenated  dictionary  D  =  [  A  B  ]  was  obtained. 

Lemma  6  (RIC  bound  for  D).  Let  D  =  [A  B]  be  characterized  by  pa, 
Pb,  Pm,  and  Pd,  and  assume  pb  <  Pa  without  loss  of  generality.  For  each 
integer  w  >  1  the  smallest  number  8W  such  that 

(1  -  8W)  ||w||2  <  || Dw H2  <  (1  +  8W)  ||w||2 

holds  for  all  perfectly  w-sparse  vectors  w  G  CNa+Nb,  is  bounded  from  above 
by 

Sw  <  min  | ^pa(w  -  2)  +  wy/p*  +  P2m^  ,  Pd(w  -  i)J  .  (20) 

As  in  Section  4.2,  we  use  the  right  hand  side  of  (20)  in  combination  with 
the  recovery  condition  5W  <  0.307  [47],  to  obtain  a  recovery  guarantee  for 
signal  separation.  In  particular,  BP-SEP  guarantees  stable  recovery  if 


w  <  max 


2(0.307  +  pa)  l  |  0.307 
Pa  +  \/pl+  Pm  Pd 


(21) 
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Figure  1:  Comparison  of  the  recovery  conditions  for  =  yd  =  0.1  and  ya  =  Pb  =  0.04. 

As  in  the  case  of  BP  restoration,  this  condition  turns  out  to  be  more  re¬ 
strictive  than  that  in  (13)  for  most  relevant  cases.  However,  the  difference 
between  the  recovery  condition  in  (21)  and  that  in  (13)  is  small,  in  general. 
To  see  this,  consider  the  two-ONB  case,  i.e. ,  [ia  —  fib  —  0  and  /im  =  pd. 
Here,  the  recovery  condition  (21)  corresponds  to  w  <  0.614 /  pd,  which  is  only 
slightly  more  restrictive  than  the  condition  w  <  2/(3 /j,d)  given  in  (15). 

4-3.  Comparison  of  the  recovery  guarantees 

Fig.  1  compares  the  recovery  conditions  for  the  general  model  (1)  to  those 
obtained  in  [12,  18]  for  perfectly  sparse  signals  and  noiseless  measurements 
(see  also  Table  1).  We  set  pd  =  =  0.1  and  pa  =  p,b  =  0.04,  and  compare 

the  three  different  cases  analyzed  in  Section  3.  The  following  observations 
can  be  made: 

•  Direct  restoration:  For  DR,  no  recovery  guarantees  are  available  through 
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the  RIP  framework.  The  recovery  conditions  for  the  general  model  (1) 
detailed  in  (7)  and  that  in  [12,  Eq.  11]  for  perfectly  sparse  signals 
and  noiseless  measurements  coincide.  Hence,  the  generalization  to  ap¬ 
proximately  sparse  signals  and  measurement  noise  does  not  entail  a 
degradation  in  terms  of  the  corresponding  recovery  condition. 

•  BP  restoration:  In  this  case,  the  recovery  conditions  for  the  general 
setup  considered  in  this  paper  and  the  condition  [12,  Eq.  14]  for  per¬ 
fectly  sparse  signals  and  noiseless  measurements  also  coincide.  Hence, 
generalizing  the  results  does  not  incur  a  loss  in  terms  of  the  recov¬ 
ery  conditions.  As  mentioned  in  Section  4.2,  the  recovery  condition 
obtained  trough  the  RIP  framework  turns  out  to  be  more  restrictive. 

•  BP  separation:  In  this  case,  we  see  that  all  of  the  recovery  conditions 
differ.  In  particular,  the  condition  [18,  Eq.  13]  for  perfectly  sparse 
signals  and  noiseless  measurements  is  less  restrictive  that  that  for  the 
general  case  (1).  Furthermore,  the  recovery  condition  (21)  obtained  in 
the  RIP  framework  turns  out  to  be  the  most  restrictive.  However,  the 
differences  between  the  three  recovery  conditions  are  rather  small. 

In  summary,  we  see  that  having  more  knowledge  on  the  support  sets  prior 
to  recovery  yields  less  restrictive  recovery  conditions.  This  intuitive  behavior 
can  also  be  observed  in  practice  and  is  illustrated  in  Section  5. 

We  finally  emphasize  that  all  of  the  recovery  conditions  derived  above  are 
deterministic  in  nature  and  therefore  conservative  in  the  sense  that,  in  prac¬ 
tice,  recovery  often  succeeds  for  sparsity  levels  nx  and  ne  much  higher  than 
the  corresponding  guarantees  indicate.  In  particular,  it  is  well-known  that 
deterministic  recovery  guarantees  are  fundamentally  limited  by  the  so-called 
square-root  bottleneck,  e.g.,  [12,  18,  50],  as  they  are  valid  for  all  dictionary 
pairs  A  and  B  with  given  coherence  parameters,  and  all  signal  and  interfer¬ 
ence  realizations  with  given  sparsity  levels  nx  and  ne.  Nevertheless,  we  next 
show  that  our  recovery  conditions  enable  us  to  gain  considerable  insights 
into  practical  applications;  i.e.,  they  are  useful  for  identification  of  appropri¬ 
ate  dictionary  pairs  (e.g.,  being  sufficiently  incoherent)  that  should  be  used 
for  signal  restoration  or  separation. 

5.  Application  Examples 

We  now  develop  two  application  examples  to  illustrate  the  main  results 
of  the  paper.  First,  we  show  that  direct  restoration,  BP  restoration,  and  BP 
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separation  can  be  used  for  simultaneous  denoising  and  declicking  of  corrupted 
speech  signals.  Then,  we  illustrate  the  impact  of  support-set  knowledge  for 
a  sparsity-based  in-painting  application. 

5.1.  Simultaneous  denoising  and  declicking 

In  this  example,  we  attempt  the  recovery  of  a  speech  signal  that  has  been 
corrupted  by  a  combination  of  additive  Gaussian  noise  and  impulse  noise. 
To  this  end,  we  corrupt  a  9.5  s  segment  (44100  kHz  and  16  bit)  from  the 
speech  signal  in  [51]  by  adding  zero-mean  i.i.d.  Gaussian  noise  and  impulse 
noise.  The  variance  of  the  additive  noise  is  chosen  such  that  the  mean- 
squared  error  (MSE)  between  the  L-dimensional  original  audio  signal  y  and 
the  noisy  version  y,  defined  as 

MSE  =  10 log10(||y  —  y || 2  / L)  , 

is  —30  decibel  (dB).  The  impulse  interference  (used  to  model  the  clicking 
artifacts  in  the  audio  signal)  is  generated  as  follows:  We  corrupt  10%  of  the 
samples  and  chose  the  locations  of  the  random  clicks,  which  are  modeled  by 
the  interference  vector  e,  uniformly  at  random.  We  then  generate  the  clicks 
at  these  locations  by  adding  i.i.d.  zero-mean  Gaussian  random  samples  with 
variance  0.1  to  the  noisy  signal  (whose  maximum  amplitude  was  normalized 
to  +1). 

Recovery  procedure.  Recovery  is  performed  with  overlapping  blocks  of  di¬ 
mension  M  =  1024.  The  amount  of  overlap  between  adjacent  blocks  is  64 
samples.  We  set  A  to  the  1024  x  1024-dimensional  DCT  matrix,  B  =  I m, 
and  perform  recovery  based  on  z  =  Ax  +  e  +  n.  The  main  reasons  for  using 
the  DCT  matrix  in  this  example  are  i)  the  speech  signal  is  approximately 
sparse  in  the  DCT  basis  and  ii)  the  mutual  coherence  between  the  identity 
and  the  DCT  is  small,  i.e.,  =  1  / x/512,  which  leads  to  less  restrictive 

recovery  conditions  (7),  (9),  and  (13).  For  all  three  recovery  methods,  we 
first  compute  an  estimate  x  of  x  (and  of  e  in  the  case  of  BP  separation) 
and  then  compute  an  estimate  of  the  denoised  speech  signal  according  to 
y  =  Ax.  In  order  to  reduce  undesired  artifacts  occurring  at  the  boundaries 
between  two  adjacent  blocks,  we  add  and  overlap  the  recovered  blocks  using 
a  raised-cosine  window  function  when  resynthesizing  the  entire  speech  signal. 
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(a)  Original  signal 


(b)  Noisy  signal  (MSE  =  -30.0 dB) 


(c)  Noise  and  clicks  (MSE  =  — 19.7dB)  (d)  Direct  restoration  (MSE  =  — 34.7dB) 


(e)  BP  restoration  (MSE  =  — 33.3dB)  (f)  BP  separation  (MSE  =  — 31.6dB) 


Figure  2:  Mean  squared  error  (MSE)  results  of  simultaneous  reduction  of  Gaussian  and 
impulse  interference  in  a  corrupted  speech  signal  (the  x-axes  correspond  to  sample  indices, 
the  y-axes  to  magnitudes). 
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Discussion  of  the  results.  Fig.  2  shows  snapshots  of  the  corruption  and  recov¬ 
ery  procedure  and  the  associated  MSE  results.  The  corrupted  signal,  which 
is  impaired  by  Gaussian  noise  and  impulse  noise,  has  an  MSE  of  — 19.7dB. 
The  individual  results  of  the  three  recovery  procedures  analyzed  in  this  paper 
are  as  follows: 

•  Direct  restoration:  In  this  case  (see  Fig.  2(d)),  the  locations  S  of  the 
impulse  noise  realizations  are  assumed  to  be  known  prior  to  recovery. 
We  set  X  =  {1, . . . ,  192},  assuming  that  the  speech  signal  occupies  only 
the  lowest  192  frequencies.  The  MSE  between  the  original  signal  and 
the  one  recovered  through  DR  corresponds  to  —34.7  dB  and,  hence,  DR 
is  able  to  improve  the  MSE  by  15  dB  (compared  to  the  signal  that  is 
impaired  by  Gaussian  and  impulse  noise). 

•  BP  restoration:  Here,  we  assume  that  the  locations  of  the  impulse 
noise  spikes  S  are  known  prior  to  recovery  but  nothing  is  known  about 
x  (except  for  the  fact  that  it  allows  for  an  approximately  sparse  rep¬ 
resentation  in  A).  We  perform  BP  restoration  with  rj  =  0.6,  which 
results  in  an  MSE  of  — 33.3dB  (see  Fig.  2(e)).  Note  that  the  parame¬ 
ter  r]  determines  the  amount  of  denoising  (for  the  Gaussian  noise)  and 
can  be  used  to  tune  the  resulting  MSE. 

•  BP  separation:  In  this  case  we  assume  that  nothing  is  known  about 
the  support  sets  of  either  x  or  e.  We  use  BP  separation  with  r/  =  0.4 
and  discard  the  recovered  error  component  e;  the  resulting  MSE  corre¬ 
sponds  to  — 31.6dB  (see  Fig.  2(f)).  As  it  is  the  case  for  BP  restoration, 
r)  determines  the  amount  of  denoising  (of  the  Gaussian  noise  compo¬ 
nent).  BP  separation  achieves  surprisingly  good  recovery  performance 
(compared  to  DR  and  BP-RES),  while  being  completely  blind  to  the 
locations  of  the  sparse  corruptions.  Hence,  BP  separation  offers  an 
elegant  way  to  mitigate  impulse  noise  in  speech  signals,  without  re¬ 
quiring  sophisticated  algorithms  that  detect  the  locations  of  the  sparse 
corruptions. 

This  application  example  shows  that  more  knowledge  on  the  support  sets 
X  and/or  S  leads  to  improved  recovery  results  (i.e. ,  smaller  MSE).  This 
intuitive  behavior  has  been  observed  in  [12]  for  perfectly  sparse  signals  and 
noiseless  measurements.  We  emphasize  that  DR,  BP  restoration,  and  BP 
separation  are  all  able  to  simulatenously  reduce  Gaussian  noise  and  impulse 
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interference  as  the  resulting  MSEs  are  all  smaller  than  —30  dB  (corresponding 
to  the  MSE  of  the  noisy  signal).  The  recovery  procedure  one  should  use  in 
practice  depends  on  the  amount  of  support-set  knowledge  available  prior  to 
recovery. 

We  finally  note  that  reduction  of  noise  and  clicks  in  impaired  audio  sig¬ 
nals  is  a  well-studied  topic  in  the  literature  (see,  e.g.,  [2]  and  references 
therein).  However,  most  of  the  established  methods  rely  on  Bayesian  es¬ 
timation  techniques,  e.g.,  [2,  52,  53],  which  differ  from  the  sparsity-based 
approach  proposed  here.  Sparsity-based  algorithms  for  restoration  of  im¬ 
paired  audio  signals  have  been  proposed  recently  in  [16,  17];  however,  no 
recovery  guarantees  are  available  for  the  proposed  OMP-based  restoration 
algorithm.  We  emphasize  that  virtually  all  proposed  methods  require  knowl¬ 
edge  of  the  locations  of  the  sparse  corruptions  prior  to  recovery,  whereas  our 
results  for  BP  separation  show  that  sparse  errors  can  effectively  be  removed 
blindly  from  speech  signals.  The  main  goal  of  this  example  is  to  illustrate 
the  performance  our  algorithms  and  not  to  benchmark  the  performance  rela¬ 
tive  to  existing  methods;  a  detailed  performance  and  restoration-complexity 
comparison  with  existing  methods  for  simultaneous  denosing  and  declicking 
is  left  for  future  work. 

5.2.  Removal  of  scratches  in  old  photographs 

We  now  consider  a  simple  sparsity-based  in-painting  application.  While 
a  plethora  of  in-painting  methods  have  been  proposed  in  the  literature  (see, 
e.g.,  [6,  8-10]  and  the  references  therein),  our  goal  here  is  to  not  to  benchmark 
our  performance  vs.  theirs  but  rather  to  illucidate  the  differences  between 
BP  restoration  and  BP  separation,  i.e.,  to  quantify  the  impact  of  support- 
set  knowledge  on  the  recovery  performance.  In  this  particular  example,  we 
intend  to  remove  scratches  from  old  photographs.  To  this  end,  we  corrupt  a 
512  x  512  greyscale  image  by  adding  a  mask  containing  artificially  generated 
scratch  patterns.  This  procedure  corrupts  15%  of  the  image.  In  order  to 
demonstrate  the  recovery  performance  for  approximately  sparse  signals,  the 
image  was  not  sparsihed  prior  to  adding  the  corruptions  (which  is  in  contrast 
to  the  in-painting  example  shown  in  [12]).  The  MSE  between  the  original 
photo  (shown  in  Fig.  3(a))  and  the  corrupted  version  (shown  in  Fig.  3(b)) 
corresponds  to  —16.7  dB. 

Restoration  procedure.  Scratch  removal  proceeds  as  follows.  For  BP  restora¬ 
tion  and  BP  separation,  we  assume  that  the  image  admits  an  approximately 
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(a)  Original  (courtesy  of  NASA  [54]) 


(b)  Corrupted  photo  (MSE  =  — 16.7dB) 


(c)  Result  of  BP-RES  (MSE  =  -29.2  dB)  (d)  Result  of  BP-SEP  (MSE  =  -19.4dB) 

Figure  3:  Example  of  using  BP  restoration  (BP-RES)  and  BP  separation  (BP-SEP)  for 
removal  of  scratches  from  old  photographs. 
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sparse  representation  in  the  two-dimensional  DCT  basis  A,  whereas  the  in¬ 
terference  is  assumed  to  be  sparse  in  the  identity  basis  B  =  I m-  Recovery 
is  performed  on  the  basis  of  the  full  512  x  512  pixel  image,  i.e.,  we  have 
M  =  5122  corrupted  measurements.  For  BP  restoration,  we  assume  that 
the  locations  of  the  scratches  are  known  prior  to  recovery,  whereas  no  such 
knowledge  is  required  for  BP  separation.  For  BP  restoration  we  recover  x 
(for  BP  separation  we  additionally  recover  e)  and  then  compute  an  estimate 
of  the  image  as  y  =  Ax.  Since  we  consider  noiseless  measurements,  we 
set  77  =  0  for  both  recovery  procedures.  Note  that  DR  is  not  considered  in 
this  example  as  information  on  the  location  of  the  dominant  entries  of  x  is 
difficult  to  acquire  in  practice. 

Discussion  of  the  results.  Fig.  3  shows  results  of  the  corruption  and  recov¬ 
ery  procedure  along  with  the  associated  MSE  values.  For  BP  restoration, 
we  see  that  the  recovered  image  has  an  MSE  of  — 29.2dB  and  well  approx¬ 
imates  the  ground  truth.  For  BP  separation,  the  MSE  improves  over  the 
corrupted  image,  but  in  parts  where  large  areas  of  the  image  are  corrupted, 
blind  removal  of  scratches  fails.  Hence,  knowing  the  locations  of  the  sparse 
corruptions  leads  to  a  significant  advantage  in  terms  of  MSE  and  is  therefore 
highly  desirable  for  sparsity-based  in-painting  methods. 

We  finally  note  that  the  recovery  conditions  (7),  (9),  and  (13)  turn  out  to 
be  useful  in  practice  as  they  show  that  the  dictionary  A  must  both  i)  sparsify 
the  signal  to  be  recovered  and  ii)  be  incoherent  with  the  interference  dictio¬ 
nary  B.  Note  that  the  second  requirement  is  satisfied  for  the  DCT-Identity 
pair  used  here,  whereas  other  transform  bases  typically  used  to  sparsify  im¬ 
ages  (i.e.,  to  satisfy  the  first  requirement),  such  as  wavelet  bases,  exhibit 
high  mutual  coherence  with  the  identity  basis.  Hence,  our  recovery  guaran¬ 
tees  help  to  identify  good  dictionary  pairs  for  a  variety  of  restoration  and 
separation  problems. 

6.  Conclusions 

In  this  paper,  we  have  generalized  the  results  presented  in  [12,  18]  for 
the  recovery  of  perfectly  sparse  signals  that  are  corrupted  by  perfectly  sparse 
interference  to  the  much  more  practical  case  of  approximately  sparse  sig¬ 
nals  and  noisy  measurements.  We  proposed  novel  restoration  and  separation 
algorithms  for  three  different  cases  of  knowledge  on  the  location  of  the  dom¬ 
inant  entries  (in  terms  of  absolute  value)  in  the  vector  x,  namely  1)  direct 
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restoration,  2)  BP  restoration,  and  3)  BP  separation.  Moreover,  we  devel¬ 
oped  deterministic  recovery  guarantees  for  all  three  cases.  The  application 
examples  demonstrated  that  our  recovery  guarantees  explain  which  dictio¬ 
nary  pairs  A  and  B  are  most  suited  for  sparsity-based  signal  restoration  or 
separation.  Our  comparison  of  the  presented  deterministic  guarantees  with 
similar  ones  obtained  using  the  restricted  isometry  property  (RIP)  frame¬ 
work  and  to  those  provided  in  [12,  18]  reveals  that,  for  BP  restoration  and 
BP  separation,  considering  the  general  model  does  not  result  in  more  restric¬ 
tive  recovery  conditions.  For  BP  separation,  however,  the  recovery  conditions 
for  the  general  model  considered  here  turn  out  to  be  slightly  more  restrictive 
as  it  is  for  perfectly  sparse  signals  and  noiseless  measurements. 

There  are  many  avenues  for  follow-on  work.  The  derivation  of  proba¬ 
bilistic  recovery  guarantees  (with  randomness  in  the  signals  rather  than  in 
the  dictionaries)  leading  to  recovery  conditions  guaranteeing  restoration  and 
separation  with  high  probability  is  an  interesting  open  research  topic.  Fur¬ 
thermore,  a  detailed  exploration  of  more  real-world  applications  using  the 
restoration  and  separation  techniques  analyzed  in  this  paper  is  left  for  future 
work. 

Appendix  A.  Proof  of  Theorem  1 

The  proof  follows  closely  that  given  in  [31,  Thm.  2.1]  and  relies  on  tech¬ 
niques  developed  earlier  in  [20,  31,  43]. 

Appendix  A.l.  Prerequisites 

We  start  with  the  following  definitions.  Let  h  =  x  —  x,  where  x  denotes 
the  solution  of  BPDN  and  x  is  the  vector  to  be  recovered.  Furthermore, 
define  ho  =  P.vh  with  the  set  X  =  suppn  (x).  The  proof  relies  on  the 
following  facts. 

Cone  constraint.  Let  eo  =  2 ||x  —  x^l^  with  x^>  =  P^x;  then  [20,  43,  46] 

||h  —  h0 1| x  <  1 1 h0 1 1 x  +  e0  (A.l) 

which  follows  from  the  fact  that  BPDN  delivers  a  feasible  solution  x  satisfying 
||xlli  >  ||x||i  and  from 

||x|| x  >  ||x|| x  =  ||xA>||1  +  ||x^c||1  =  llx*  +  h0 1| x  +  || h  -  h0  +  h^cl^ 

>  1 1 x^- 1 1 x  -  Hholli  +  || h  -  holli  -  ||x;Hli  • 
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Application  of  the  reverse  triangle  inequality  to  the  left-hand  side  term 
of  (A.l)  yields  the  following  useful  bound: 


||li|| x  <  2||ho||1  +  eo-  (A. 2) 

Tube  constraint.  We  furthermore  have  [43,  46] 

|| Ah|| 2  =  || Ax  -  y  -  (Ax  -  y)||2 

<  ||Ax-y||2  +  || Ax  —  y||2  <  p  +  e.  (A.3) 

Coherence-based  restricted  isometry  property  (RIP).  Since  h0  is  perfectly  nx- 
sparse,  Gersgorin’s  disc  theorem  [55,  Thm.  6.1.1]  applied  to  ||  Aho || 2  yields 

(1  -  fia(nx  -  1) ) || h0 1| 2  <  II  Ah0 1| 2  <  (1  +  Ra{nx  -  1)) ||h0||2  •  (A.4) 

Appendix  A. 2.  Bounding  the  error  ||ho||2  on  the  signal  support 

The  goal  of  the  following  steps  is  to  bound  the  recovery  error  ||h0||2  on 
the  support  set  X.  We  follow  the  steps  in  [31]  to  arrive  at  the  following  chain 
of  inequalities: 

jh^A^Ahol  >  jh^A^Ahol  -  Kh-ho^A^Ahol 

>  (1  -  na(nx  -  1))  ||h0||2  -  lho]k^kMh}e  (A-5) 

kex  iexc 

>  (1  -  pa(nx  -  1))  ||h0||2  -  pa 1 1 h0 1 1 x  || h  -  hdh  (A.6) 

>  (1  -  Ra(nx  -  1))  ||h0||2  -  p<a || h0 1| x  (Hholli  +  e0)  (A.7) 

—  (^  1))  1 1  -h-0 1 1 2  I^cl^x  ||  h-Q  ||  2  \J~^x  \ |  h-0  1 1 2  ^0 

(A.8) 

=  (1  -  na(2nx  -  1))  ||h0||2  -  HaVrc\\ho\\2  eo,  (A.9) 

where  (A-5)  follows  from  (A.4),  (A.. 6)  is  a  consequence  of  ja^a^l  <  /za,  \/k  ^  £, 
(A. 7)  results  from  the  cone  constraint  (A.l),  and  (A.8)  from  the  Cauchy- 
Schwarz  inequality.  We  emphasize  that  (A. 9)  is  crucial,  since  it  determines 
the  recovery  condition  for  BPDN.  In  particular,  if  the  first  RHS  term  in  (A. 9) 
satisfies  (1— /ia(2nx  —  1))  >  0  and  h0  ^  0AraXi,  then  the  error  || h0 1| 2  is  bounded 
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from  above  as  follows: 


^  |h"A"Ah0|  +/iax/n^||h0||2e0 

°"2  “  (l-M2n*-l))||ho||2 

<  II  Ah||2  ||  Ahpl|2  +  /Xay^||ho||2  e0 

(1  -^o(2nx  -  1))  ||ho||2 

<  (g  +  77) y/1  +  fj.g(nx  -  l~y 1 1 h0 1 [ 2  +  /jax/n^ ||ho||2  e0 

(1  -^a(2nx-  1))  ||ho||2 

_  (e  +  v)y/ 1  +  Ma(nx  ~  1)  +  ha\Ane0 
1  -  na(2nx  -  1) 


(A.10) 

(A.ll) 

(A.12) 

(A.13) 


Here,  (A.  10)  is  a  consequence  of  (A. 9),  (A.ll)  follows  from  the  Cauchy- 
Schwarz  inequality,  and  (A.12)  results  from  the  tube  constraint  (A. 3)  and 
the  RIP  (A. 4).  The  case  ho  =  OjvaXi  is  trivial  as  it  implies  ||ho||2  =  0. 


Appendix  A. 3.  Bounding  the  recovery  error  ||h||2 

We  are  now  ready  to  derive  an  upper  bound  on  the  recovery  error  ||h||2. 
To  this  end,  we  first  bound  ||Ah||2  from  below  as  in  [31] 


Ah||  2  =  h"A"Ah  =  y][hH]fea^ai[h]g 

k/ 


=  X]llafcll2  l[h]fc|2  +  [hH]kakMh]e 

k  k,t,k+t 

>  IlhUa  -  ha 

k/,k^e 


=  \Ml  +  ha^2\[h]k\2  -paJ2\[hH}k[h]e 

k  k,t 

=  (1  +  fV)llhll2  -ft.llhll?  . 


(A. 14) 


(A. 15) 


where  (A.14)  follows  from  ||ar||2  —  1,  Vfc,  and  aj^ad  <  (ia,  v'/,:  /  i  With 
(A.  15),  the  recovery  error  can  be  bounded  as 

II h II 2  <  l|Ah||2  +  /A0||h||j 

2  1  +  Ra 

^  (£  +  v)2  +  Ba  (2||h0|| !  +  e0)2  ,A 


28 


where  (A. 2)  is  used  to  arrive  at  (A.  16).  By  taking  the  square  root  of 
(A.  16)  and  applying  the  Cauchy- Schwarz  inequality,  we  arrive  at  the  fol¬ 
lowing  bound: 


l|h||2< 

< 


\j{£  +  V)2  +  ha  (2||h.0|| x  +  e0)2 
a/1  +  ha 

(e  +  rj)  +  ^fJTa  (211^1^  +  ep) 
\/l  +  ha 


(A.  17) 


Finally,  using  || h0 1| x  with  the  bound  in  (A. 13)  followed  by  alge¬ 

braic  simplifications  yields 


lhll2  < 


(s  +  rj)  +  yfjla  (2v^l|ho||2  +  eo) 


<  (e  +  rj) 
+  Cq 


\/l  + 

f  ha(2ua;  1)  T  2^/p,anx^/l  -|-  iia(n x  f) 

V1  +ha  (1  -  Hai^rix  -  1)) 

\/ha  +  ha 


(1  -  ^(2'^  -  1)) 

=  Co(7?  +  e)  +  Ci||x  -  x^lli , 


which  concludes  the  proof.  We  note  that  by  imposing  a  more  restrictive 
condition  than  nx  <  (1  +  l//xa)/2  in  (2),  one  may  arrive  at  smaller  constants 
C0  and  C\  (see  [46]  for  the  details). 


Appendix  B.  Proof  of  Theorem  2 

The  proof  is  accomplished  by  deriving  an  upper  bound  on  the  resid¬ 
ual  errors  resulting  from  direct  restoration.  Furthermore,  we  show  that  the 
recovery  condition  (7)  guarantees  the  existence  of  =  I m  —  B^Bjr  and 
(R*A*)t. 

Appendix  B.l.  Bounding  the  recovery  error 

We  start  by  bounding  the  recovery  error  of  DR  as 

||x  -  x||2  <  ||x*  -  X^  ||  2  +  ||x*c  -  X^c||2 

<  ||xAf  -  X*||2  +  ||x^c  ||  x  .  (B.l) 
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The  only  term  in  (B.l)  that  needs  further  investigation  is  ||x*  —  Xv||2-  As 
shown  in  (5),  we  have 

[x]v  =  (R.fA^'I’Rgz  =  x#  +  (RgAx^R^Ax^c  +  n) 
and  hence,  it  follows  that 


||x*— x*||2  <  ||  (Rc;Aa’)^R£-v||2  ,  (B.2) 

where  v  =  Ax^c  +  n  represents  the  residual  error  term.  The  remainder  of 
the  proof  amounts  to  deriving  an  upper  bound  on  the  RHS  in  (B.2).  We 
start  with  the  definition  of  the  pseudo-inverse 

||(R£A.»)tR£v||2  =  |j(A.?R£A.,.)-1A.?R£v|[2  (B.3) 

<  ||(AjR£AU_1||2||AjR€v||2,  (B.4) 

where  (B.3)  is  a  consequence  of  R^-  =  R^Rg,  and  (B.4)  follows  from  the 
Rayleigh- Ritz  theorem  [55,  Thm.  4.2.2],  We  next  individually  bound  the 
RHS  terms  in  (B.4)  from  above. 


Appendix  B.2.  Bounding  the  £2-norm  of  the  inverse 

The  bound  on  the  norm  of  the  inverse  in  (B.4)  is  based  upon  an  idea 
developed  in  [24],  Specifically,  we  use  the  Neumann  series  (I|*|  —  K)-1  = 
I\x\  +  Y^k= i  [56,  Lem.  2.3.3]  to  obtain 


(A^RfA 


x 


(AfA^-AfB^A*)- 

=  ll(iw-K)-l||2=  l\x\+J2Kk 


k= 1 


<  1  + 


k= 1 


<i+DKi 


k= 1 


|k||2’ 


(B.5) 


which  is  guaranteed  to  exist  whenever  ||K  II 2  <  1  with 


K  —  I|v|  —  A^Ax  +  A^BcB^A^. 
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We  next  derive  a  sufficient  condition  for  which  ||K||2  <  1  and,  hence,  the 
matrix  A^Rf-A^  is  invertible.  We  bound  ||K||2  from  above  as 


K||2<  1\x\  -  A"A 


H 


x^x\\2 


+ 


A%B£BlA 


A” 


<Ha(nx-  1)  +  A"Bf(B"B 


\  — Itd  H 


B"  Ax 


2  ’ 


(B.6) 

(B.7) 


where  (B.6)  results  from  the  triangle  inequality  and  (B.7)  is  a  consequence 
of  Gersgorin’s  disc  theorem  [55,  Thm.  6.1.1]  applied  to  the  G-norm  of  the 
hollow  matrix  —  A^Ax-  We  next  bound  the  RHS  term  in  (B.7)  as 

WAxBeiBgBer'B* Ax\\2  <  \\A%Be\\2  ||(B*Be)-1||2  \\B% Ax\\2  (B.8) 

<  ||A*Be||*  ||(B£Be)-1||2  (B.9) 

<n*ne/£||(B"Be)-1||2  (B.10) 


< 


nxnen2m  < 
Amin(BfB£)  “ 


nxnen ^ 

[l-/x&(ne-l)]+’ 


(B.ll) 


where  (B.8)  follows  from  the  f^-matrix-norm  bound,  (B.9)  from  ||A^Bf||  < 
||A^B£||f  and  ||A£B£||f  =  ||B^A*||f,  and  (B.10)  from 


AVB^||f  =  ^2^2\akhe\2  <J2J2^=nxn^2rr 
kex  ie£  kex  ie£ 


Note  that  (B.ll)  requires  ne  <  1  +  1/  fib,  which  provides  a  sufficient  condition 
for  which  the  pseudo-inverse  Bt  exists. 

Combining  (B.5),  (B.7),  and  (B.ll)  yields  the  upper  bound 


(a"rjau-1||2<  - - — 

1 


which  requires 


1 

1  \  _ 

'  [l-/ii,(ne-l)]  + 


(B.12) 


1  -  Va(nx  -  1) 


nxnen2m 

[l-^(ne-l)]  + 


>  0 


(B.13) 


for  the  matrix  (A^RcA^)  1  to  exist.  We  emphasize  that  the  condition 
(B.13)  determines  the  recovery  condition  for  DR  (7).  In  particular,  if 


(1  -  fj,a(nx  -  1))  [1  -  Hb(ne  -  1)]+  >  nxnen2m 


then  (B.13)  and  ne  <  1  +  1/fib  are  both  satished  and,  hence,  the  recovery 
matrix  (R^A,^) '  Rg  required  for  DR  exists. 
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Appendix  B.3.  Bounding  the  residual  error  term 

We  now  derive  an  upper  bound  on  the  RHS  residual  error  term  in  (B.4) 
according  to 


AvRfV  _  <  A?v  .  + 


H, 


*v|l2 


<  \fn~x II v 


A^BcBcV 

A^BcB^v 


where  (B.14)  is  a  result  of 


avvI|2  =  J^2\akv\2  <  IMI2  <  VrkclN 


2  • 


(B.14) 


(B.15) 


kex 


kex 


The  bound  on  the  second  RHS  term  in  (B.14)  is  obtained  by  carrying  out 
similar  steps  used  to  arrive  at  (B.ll),  i.e., 


A^BcBtv 


<  l|A^B£||2||(BfB£)-1||2||Bfv 

<  Vnxnel^m  ll-pjf  || 

-  Amin(BfB£) 11  £  112 

ne  sJ'klxV^n  II  || 

<  - - - x  vL. 

[1  -  nb(ne  -  1)]  + 


(B.16) 


Finally,  we  bound  the  £2-norm  of  the  residual  error  term  according  to 


=  HAx 


vc 


n 


I2  — 


I  Ax 


vc 


+  ||n||2  <  ||x^c 


n 


(B.17) 


since 


I  Ax 


*cll2  - 


E 

kexc 


a*  xk 


< 


E 

kexc 


ak  x  R  2  =  x^c 


Appendix  B.4-  Putting  the  pieces  together 

In  order  to  bound  the  recovery  error  on  the  support  set  A,  we  combine 
(B.12)  with  (B.14)  and  (B.16)  to  arrive  at 


(R*A*)tRfV  < 


(  1  fi'b(p,e  1)]  T 


(1- 

=  cllvl 


Pa(nx  -  1))  [1  -  p,b(ne  -  1)]+  -  nxnetf 
2  • 


(B.18) 
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Finally,  using  (B.17)  in  combination  with  (B.l),  (B.2),  and  (B.18)  leads  to 

||x  -  x||2  <  C£  +  (c  +  1)||X  -  ~X-X  ||  \ 

=  C3£  +  C4||x  -  X^f Hi  , 

which  concludes  the  proof. 

Appendix  C.  Proof  of  Theorem  3 

We  first  derive  a  set  of  key  properties  of  the  matrix  A  =  R^A,  which  are 
then  used  to  proove  the  main  result  following  the  steps  in  Appendix  A. 

Appendix  C.l.  Properties  of  the  matrix  A 

It  is  important  to  realize  that  BP  restoration  operates  on  the  input-output 
relation 

Rgz  =  Rg  (Ax  +  Be,*-  +  n)  =  Ax  +  R^-n  (C.l) 

with  Rg  =  I m  —  BcB^.  and  A  =  RgA.  The  recovery  condition  for  BP 
restoration  (9),  which  will  be  derived  next,  also  ensures  that  Rf-  exists;  this 
is  due  to  fact  that  the  recovery  condition  for  DR  (7)  ensures  that  R^  exists 
and  the  condition  for  BP  restoration  (9)  is  met  whenever  (7)  is  satisfied. 

In  order  to  adapt  the  proof  in  Appendix  A  for  the  projected  input-output 
relation  in  (C.l),  the  following  properties  of  A  are  required. 

Tube  constraint.  Analogously  to  Appendix  A,  we  obtain 

Ah  <  ||Rf(Ax  -  z) || 2  +  ||R£(Ax  -  z)||2 

<  77+  ||Rfn||2  <  v  +  e, 

where  the  last  inequality  follows  from  the  fact  that  Rg  is  a  projection  matrix 
and,  hence,  ||Rfn||2  <  || n|| 2  <  e. 

Coherence-based  bound  on  the  RIC.  We  next  compute  a  coherence-based 
bound  on  the  RIC  for  the  matrix  A.  To  this  end,  let  ho  be  perfectly  nx- 
sparse  and 

Ah0  ^  =  h^A^ Ah0  -  h^AHBcBjAh0  (C.2) 

<  (1  +  Ha{nx  ~  1)) II h0 1| 2  +  |h^A^B,BjAh0|  ,  (C.3) 
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where  (C.2)  follows  from  Rj^R.f  =  Rg  and  (C.3)  from  Gersgorin’s  disc  the¬ 
orem  [55,  Thm.  6.1.1].  Next,  we  bound  the  second  RHS  term  in  (C.3)  as 
follows: 


h"A',B£(Bj'B£)-1B|'Ah(l 


<  Am;n(B»B£)||BfAh0||' 

<Ajn(Bj'B£)||B|'A*||*||h0||l 


< 


[l-lib(ne-l))  + 


(C.4) 

(C.5) 

(C.6) 


where  (C.4)  follows  from  [55,  Thm.  4.2.2],  (C.5)  from  the  -G-norm  inequality. 
The  inequality  (C.6)  results  from 

||Bf  Ax\\l  <  ||Bf  Ax\\2f  =  5^lb^afc|2  -  n^nA 

e&£  k£X 


Note  that  (C.6)  requires  ne  <  1  +  l/fib,  which  is  a  sufficient  condition  for 
(B^Bf)'1  to  exist.  Note  that  ne  <  1  +  1/fib  holds  whenever  the  recovery 
condition  for  BP-RES  in  (7)  is  satisfied.  Combining  (C.3)  with  (C.6)  results 
in 


RfAhollg  <  (  1  +  fiai^x  —  1)  + 


nxnefi ‘ 


[1  -  fib(ne  -  1)]' 


a0||2 


=  (!  +  ■ 


m2 


(C.7) 


We  next  compute  the  lower  bound  as 
1 2 


Ahn 


Ahn  -  h^A^BcBtAh, 


0 


■>£D£1 


>  (1  -  Ha{nx  -  1) ) || h0 1| 2  - 


h,f  AflBcBjrAhc 


>  (1  -  fia(nx  -  1) ) || h0 1| 2  - 


nxnefi ~ 


[1  -  fib(ne  -  1)] 


+  llI1o||2 


>  1  -  fia(nx  -  1) 


nxnefi * 


[1  -  fib(ne  -  1)]' 


ao||2 


=  (l-tf)||ho| 


(C.8) 

(C.9) 

(C.10) 


where  (C.8)  follows  from  [55,  Thm.  6.1.1]  and  (C.9)  is  obtained  by  carrying 
out  similar  steps  used  to  arrive  at  (C.6).  Note  that  (C.7)  and  (C.10)  provide 
a  coherence-based  upper  bound  5  on  the  RIC  of  the  projected  matrix  A  = 

Rf  A. 


34 


Upper  bound  on  the  inner  products.  The  proof  detailed  in  Appendix  A  re¬ 
quires  an  upper  bound  on  the  inner  products  of  columns  of  the  matrix  A. 
For  i  ^  j,  we  obtain 


afa j  =  a''Rca;  <  a" a,  + 


H 


H , 


afBcB  Uj 


<pa+  |aiB£(BfB£)  | 

lafBpB^aJ 

<  Pa  + 


afBcBjS 

[1  -  Hb{ne  -  1)]~ 


<  ha 


lBM2 


bJSH2 


[1  -  pb(ne  -  1)] 


+  > 


(C.ll) 

(C.12) 

(C.13) 


where  (C.ll)  follows  from  the  definition  of  the  coherence  parameter  pa, 
(C.12)  is  a  consequence  of  Gersgorin’s  disc  theorem,  and  (C.13)  from  the 
Cauchy- Schwarz  inequality.  Since 


|Bfa, 


'Elbfa,|2  <  V' 


nehm 


for  all  Na,  the  inner  products  with  i  ^  j  satisfy 


af  hj  <  \ia  + 


nehm 

[1  -  hb(ne  -  1)]' 


A 

=  a. 


(C.14) 


Lower  bound  on  the  column  norm.  The  last  prerequisite  for  the  proof  is  a 
lower  bound  on  the  column- norms  of  A.  Application  of  the  reverse  triangle 
inequality,  using  the  fact  that  1 1 a,; 1 1 2  =  1,  Vi,  and  carrying  out  the  similar 
steps  used  to  arrive  at  (C.14)  results  in 


y2  =  HBfA'iio  >  afa* 


ifB.cBf 


£ai 


>  1 


n-ehm 


[1  -  Hb(jie  -  1)]' 


^  b. 


Appendix  C.2.  Recovery  guarantee 

We  now  derive  the  recovery  condition  and  bound  the  corresponding  error 
||h||2.  The  proof  follows  that  of  Appendix  A.  For  the  sake  of  simplicity  of 
exposition,  we  make  use  of  the  previously  defined  quantities  5,  a,  and  b. 
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Bounding  the  error  on  the  signal  support.  We  start  by  bounding  the  error 
|| ho || 2  as  follows: 

h"A"Ah0 

>  (1  -  A) ||hQ H2  -  an^Hholl^  -  a^/n^Wh^eo 
c||h0||2  av'n*  ||  ho  ||  2  0 


> 


h^A^Ahc 


(h-ho)HAHAhc 


(C.15) 


with 


c  —  1  —  h  —  anx  =  1  -  Ha(?nx  -  1)  - 


2  nxnen: 


[l-/ib(ne-l)]  + 

Note  that  the  parameter  c  is  crucial,  since  it  determines  the  recovery  condi¬ 
tion  for  BP-RES  (9).  In  particular,  c  >  0  is  equivalent  to  (9) 

[1  -  Ha(2nx  -  1)]+  [1  -  pb(ne  -  1)]+  >  2nxnegL2m. 

If  this  condition  is  satished,  then  we  can  bound  ||h0||2  from  above  as  follows: 

M,  ^  (£  -  n)  Cl  +  S  +  a^/n^eo 
l|h0||2  <  - - - • 

Bounding  the  recovery  error.  We  next  compute  an  upper  bound  on  ||h|L.  To 


this  end,  we  start  with  a  lower  bound  on 


Ah 


as 


Ah 


>  (6  +  a)||h||2  —  a||h||^  =  (1  +  /ia)||h||2  —  a||h||^  , 


since  h  +  a  =  1  +  pa.  Finally,  we  bound  h±±h2 


as  follows: 


hi  11  ^  /  .  n.  c  +  2^/ anx \/ 1  +  <5  y/ay/l  +  pa 

l|h|'2  -  (£  +  V)  V1  +  M.C  +  eo - c - 

=  C5(rj  +  e)  +  C6||x  -  Xxlh  , 

where  the  constants  C5  and  Cq  depend  on  pa,  pb,  nx,  and  ne,  which  concludes 
the  proof. 


Appendix  D.  Proof  of  Theorem  4 

In  order  to  prove  the  recovery  guarantee  in  Theorem  4,  we  start  by 
deriving  a  coherence-based  bound  on  the  RIC  of  the  concatenated  matrix 
D  =  [  A  B]  which  is  then  used  to  prove  the  main  result. 
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Appendix  D.l.  Coherence-based  RIC  for  D  =  [  A  B] 

In  this  section,  we  obtain  an  equivalent  bound  to  that  in  Appendix  A.l 
for  the  dictionary  D  that  depends  only  on  the  coherence  parameters  fia,  /q,, 
Hm,  and  fid,  and  the  total  number  of  nonzero  entries  denoted  by  w  =  nx  +  ne. 

Bounds  that  are  explicit  in  nx  and  ne.  Let  h0  =  [h^  hf]T  where  hT  = 
P*(x  —  x)  and  he  =  Pc(e  —  e)  are  perfectly  nx  and  ne  sparse,  respectively. 
We  start  by  the  lower  bound  on  the  squared  f^-norm  according  to 


|Dh0||l  =  [  hf  h«  } 


=  h. 


H 


^Na 

0 


>  IMa 


'  AHA  AhB 

bha  bhb 

0 

I  Nb 

'  AvAv - I 


H 


h0  +  h0 


h* 

he 

A"  A  -  INa 

BhA 


^X-^X  —  *-\x\ 

*2A£ 


A£Bf 


B?BS 


l|£| 


A^B 

BHB-INb 


h0 

(D.l) 


where  (D.l)  follows  from  the  reverse  triangle  inequality  and  elementary  prop¬ 
erties  of  the  £ 2  matrix  norm.  We  next  compute  an  upper  bound  on  the  matrix 
norm  in  (D.l)  as  follows: 


A%  Ax  —  I\x\  0 

0  B^B^  — 1|£| 

<  max{||AfA^ -I|^|||2, 


0 

B»AS 


Bf  Be  -  I|£j 


AfB  £  1 

0  J  2 

2}  +  IIAvB£ 


2  ’ 


(D.2) 


where  (D.2)  is  a  result  of  the  triangle  inequality  for  matrix  norms  and  the 
facts  that  the  spectral  norm  of  both  a  block-diagonal  matrix  and  an  anti- 
block-diagonal  matrix  is  given  by  the  largest  among  the  spectral  norms  of 
the  individual  nonzero  blocks.  The  application  of  Gersgorin’s  disc  theorem 
to  the  max{-}-term  in  (D.2)  and 


AfB£ 


2  — 


a£b£ 


'V'  y^la^b 


Z _ /  1  K 


'£  ^ 


< 


nrn, 


Am 


leads  to 


max{||AfA^-Iw||2,  ||BfBg  -I|£|||2}  +  ||A£B£||2 

<  max  {fia(nx  -  1),  fib{ne  -  1)}  +  \Znxnefi2m. 
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Hence,  we  arrive  at  the  following  lower  bound 


II Dh0 1| 2  >  Hholla  (l  -  max  {ga(nx  -  1  ),/ub(ne  -  1)}  -  yj nxne^  .  (D.3) 

By  performing  similar  steps  used  to  arrive  at  (D.3)  we  obtain  the  upper 
bound 

l|Dh0 1| 2  <  II h0 1| 2  +  max  {/ ia(nx  -  1  ),/xb(ne  -  1)}  +  ^nxneg2m )  .  (D.4) 

Bounds  depending  on  w  =  nx  +  ne.  Both  bounds  in  (D.3)  and  (D.4)  are 
explicit  in  nx  and  ne.  Since  the  individual  sparsity  levels  nx  and  ne  are 
unknown  prior  to  recovery,  a  coherence-based  RIP  bound,  which  depends 
solely  on  the  total  number  w  —  nx  +  ne  of  nonzero  entries  of  h0  rather  than 
on  nx  and  ne ,  is  required.  To  this  end,  we  dehne  the  function 

g(nx,ne)  =  rna x{fxa(nx  -  l),//6(ne  -  1)}  +  a Jnxneg2m 

and  hnd  the  maximum 


g[w)  =  max  g(nx,w  —  nx).  (D.5) 

0  <nx<w 

Since  g(w)  only  depends  on  w  =  nx+ne  and  g(nx,  ne)  <  g(w),  we  can  replace 
g(nx,ne)  by  g(w)  in  both  bounds  (D.3)  and  (D.4). 

We  start  by  computing  the  maximum  in  (D.5).  Assume  ga{nx  —  1)  > 
Rf>(w  —  1)  and  consider  the  function 

9a{nx,  w  -  nx)  =  na(nx  -  1)  +  y/nx(w  -  nx)g2m.  (D.6) 

It  can  easily  be  shown  that  ga(nx,  w  —  nx )  is  strictly  concave  in  nx  for  all  0  < 
nx  <  w  and  0  <  w  <  oo  and,  therefore,  the  maximum  is  either  achieved  at 
a  stationary  point  or  a  boundary  point.  Standard  arithmetic  manipulations 
show  that  the  (global)  maximum  of  the  function  in  (D.6)  corresponds  to 

(AM  =  \  (/D(w  -  2)  +  wyjg2  +  /4)  .  (D.7) 

For  the  case  where  /iQ  (nx  —  1)  <  /i;, (ne  —  1),  we  carry  out  similar  steps  used 
to  arrive  at  (D.6)  and  exploit  the  symmetry  of  (D.5)  to  arrive  at 

gbM  =  ^  ^ lb(w  -  2)  +  WyJfll+fM 2m 
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Hence,  by  assuming  that  /if,  <  /ia,  we  obtain  upper  and  lower  bounds  on 
(D.3)  and  (D.4)  in  terms  of  w  =  nx  +  ne  with  the  aid  of  (D.7)  as  follows: 

(1  -  <?a(w))||ho||2  <  II Dh0 1| 2  <  (1  +  ^oH)||h0||2  .  (D.8) 

It  is  important  to  realize  that  for  some  values  of  /ia,  /im,  and  w,  the 
bounds  in  (D.8)  are  inferior  to  those  obtained  when  ignoring  the  structure 
of  the  concatenated  dictionary  D,  i.e., 

(1  -  fid(w  -  1) ) II h0 1| 2  <  HDholla  <  (1  +  Rd(w  -  l))||h0||2  •  (D.9) 

with  /id  —  max{/ia, /i^, /im}.  However,  for  w>2)  (irn  —  and 


l^a  ^ 


m 


2 


5 


the  RIP  considering  the  structure  of  D  in  (D.8)  turns  out  to  be  more  tight 
than  (D.9).  For  other  values  of  w  and/or  fj,a,  (D.8)  turns  out  to  be  less  tight 
than  (D.9).  In  order  to  tighten  the  RIP  in  both  cases,  we  consider 

^1  —  Sw^j  1 1 h0 1 1 2  <  || Dho || 2  5;  (t  +  \\ho\\l , 

where  the  coherence-based  upper  bound  on  the  RIC  of  the  concatenated 
dictionary  D  =  [  A  B  ]  corresponds  to 

Sw  =  min  |^/ia(w  -  2)  +  wyVa  +  Pm)  *  Rd{w  “  1) 

Appendix  D.2.  Recovery  guarantee 

We  now  bound  the  error  ||h||2  and  derive  the  recovery  guarantee  by  fol¬ 
lowing  the  proof  in  Appendix  A.  In  the  following,  we  only  show  the  case 
where 

\  (/h*(W  -  2)  +  ^vVa  +  Rm)  <  Rd{w  -  1). 

The  other  case,  i.e.,  where  the  standard  RIP  (D.9)  is  tighter  than  (D.8), 
readily  follows  from  the  proof  in  Appendix  A,  by  replacing  A  by  D,  pa  by 
A id,  and  nx  by  w. 
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Bounding  the  error  on  the  signal  support.  We  start  by  bounding  the  error 
||h0||2.  Since  /im  <  [id,  we  arrive  at 

|hHDHDho|  >  |h^D^Dh0|  -  |(h  -  h0)"D//Dh0| 

>  (1  -  4)||h0||2  -  AhHIholls  -  BdVw ||h0||2  e0 
=  rf|| h0 1| 2  -  /W^I|ho||2e0 

with 

d  =  1  -  Sw  -  HdW  =  1  -  ^  (jla  +  2[id  +  yjn2a  +  [i'^j  +  [la. 

It  is  important  to  note  that  d  is  crucial  for  the  recovery  guarantee  as  it 
determines  the  condition  for  which  BP-SEP  in  (13)  enables  stable  separation. 
Specifically,  if  d  >  0  or,  equivalently,  if 

2(1  +  [la) 

W  <  - 

[la  +  2 [ld  +  V  hni  +  Mm 

then  the  error  on  the  signal  support  ||ho||2  is  bounded  from  above  as 

d yj  +  [id\/we  o 

d  ' 

where  eo  =  2||w  —  wyvl^  with  W  =  suppM(w). 

Bounding  the  recovery  error.  Analogously  to  the  derivation  in  Appendix  A. 3, 
we  now  compute  an  upper  bound  on  ||h||2,  i.e., 

II Dh|| 2  >  (1  +  hrf)l|h||2  —  hrf||h||i  •  (D.10) 

Finally,  bounding  ||h||2  similarly  to  Appendix  A. 3  results  in 

M1  „  .  d  +  2y/[idw^l  +  8w  JjTd(d  +  2[idw) 

l|h|l> 5  (£ + ">  — vi +  eo  vms 

=  CV(? 7  +  e)  +  C8||w  -  Wyy || x . 

where  the  constants  Cj  and  C8  depend  on  the  parameters  pa,  fJ>t „  [im,  [id, 
and  w  =  nx  +  ne ,  which  concludes  the  proof. 


I2  — 


(e  +  rj)  \/lH- 
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